From personal computers to file servers to internetwork equipment, the need for high operational data rates continually drives system level architecture decisions in data processing systems. Designers are typically no longer concerned with optimization at the circuit level or even at the processor level, since most systems are now implemented using standardized chip sets. Thus, overall system performance often can be optimized only by careful consideration of the architecture chosen for interprocessor communication.
Historically, such systems have been based upon a bus topology to which access is granted on a time slot basis. This architecture has worked well in personal computers and other data processors wherein input/output (I/O) devices share access to the bus with memory subsystems on a time slot basis. Because the number of time slots on a bus are fixed in advance, once all such slots are allocated among the various elements, there is little room for additional expansion. Thus, a bus oriented time slot architecture does not scale well in many applications. In addition, any attached devices must arbitrate for bus access. If any device tends to xe2x80x9chogxe2x80x9d the bus, the contention for access between devices negatively impacts the performance of all devices.
Configuration of a time slot oriented system also tends to be an art rather than a science, since the determination of which specific slot is used for which specific device can also determine system performance. Often, only with detailed experience with a particular architecture can the system be tuned for optimum performance. And, configuration of such systems becomes increasingly difficult as bus clock rates increase. Depending on their implementation, maximizing the clock rate of such high-speed time division multiplexed access buses require that the parallel bus slots be physically close to each other. Otherwise, high clocking rates may result in clock skew, causing data errors to occur.
Perhaps nowhere is the demand for increased processor performance more acute at the present time than for internetwork devices. Such devices include routers, switches, and other processors for handling Internet Protocol (IP) type data traffic through a network. The demand for high-speed Internet infrastructure continues to escalate. For example, traffic on Internet backbones doubles approximately every six to nine months. The widespread adoption of internets and extranets also continues to bring further changes to the IP service infrastructure.
In most existing high-speed internetwork devices, optical physical layer signalling technology is used to interconnect devices such as switches and routers. These devices may be directly connected with optical fiber, or may be connected to an optical network that provides wavelength routing between various devices. In either case, the switches and routers use the full bandwith capacity provided by the fiber or wavelengths to statistically multiplex packets or cells, allowing what has heretofore been thought to be an efficient use of the traffic handling capacity.
However, these designs date back to a time when data traffic represented only a small fraction of the total traffic on public networks, when the strategy of carrying data over a voice circuit oriented infrastructure was cost effective and imposed few constraints. This infrastructure typically consisted of voice switches interconnected by time division multiplexing (TDM) or circuit switched networking equipment. The TDM network has been built out using digital cross-connects and network elements such as Synchronous Optical Network/Synchronous Digital Hierarchy (SONET/SDH) Add-Drop Multiplexers (ADM) that multiplex together thousands of voice circuits in order to fill the capacity of high-speed fiber optic transmission facilities. These fibers typically operate at speeds ranging from 155 megabits per second (Mbps) to 2.5 gigabits per second (Gbps), with 2.5 Gbps now prevalent on long haul backbone facilities.
As data traffic grows to dominate total capacity requirements, the limitations of using TDM technology within the network for data delivery become increasingly apparent. Because the data traffic itself is packet oriented, it cannot be optimally transported over TDM. At the network level, technologies such as Dynamic Packet Transport (DPT) recently announced by Cisco Systems, Inc. of San Jose, Calif., can be used to combine the bandwidth efficiency, service richness, and cost effectiveness of packet switched IP protocols with the bandwidth richness and proactive self-healing capability of fiber optic ring oriented structures. With DPT, bandwidth efficiency is increased through the elimination of fixed time slots and the use of a Media Access Control (MAC) layer protocol called the Spatial Re-use Protocol (SRP). This system is optimized to support scalable and survivable IP packet transport, and thus, reduces or eliminates the need for TDM equipment in IP networks.
While DPT devices running SRP protocols hold the promise of efficient routing of data traffic through a public network structure, there still remains the difficulty of optimizing the design of the internetworking devices themselves. One finds limitations in interconnect architectures for use within internetworking devices in particular, and computing devices, in general. The shortcoming is especially acute where such devices are primarily responsible for routing data traffic, such as in the case of a device like a router, and even in computing equipment such as a file server. The existing standardized bus structures such as the Industry Standard Architecture (ISA), Peripheral Component Interconnect (PCI), and other time slot based bus structures are not optimized for the transport of data packets. They do not provide maximum payload capacity, synchronization, or the ability to gracefully expand as the demand for bandwidth over a particular connection increases.
Certain recently proposed high-speed communication interfaces, such as the Gigabit Ethernet (GE) standard, provide a high-speed physical layer within the context of an Open System Interface (OSI) reference model. The GE protocols use a significant amount of the available bandwidth for signalling and for other overhead. However, such overhead is not necessary in tightly coupled system environments, such as between the processing elements within a computing system.
The present invention is a high-speed circuit interconnect structure and a tightly coupled protocol for interconnecting electronics system components such as integrated circuits, processor chips, memory chips, Application Specific Integrated Circuits (ASICs), or other modular pieces be they physically embodied at a component, board, or chassis level.
An interconnect designed in accordance with the invention uses two or more high-speed links to transport data between components. The links use a lightweight packet oriented data protocol over a very high-speed data bus.
The protocol uses embedded control characters to delineate packets and to signal other events within individually encoded data channels. The preferred method for encoding each individual data channel is based on an 8-bit to 10-bit (8 B/10 B) method of encoding that supports the embedded control characters. The control characters are used to indicate the end of a packet without error, to indicate the end of a packet with error, to synchronize the receiver-transmitter pair, and to control the flow of data generated by the transmitter. The special control characters are also used in a particular format that allows concatenation of multiple links between a particular transmitter and receiver. In this way, a given packet may be efficiently transmitted on multiple links.
One particular embodiment of the invention is within an internetworking device, such as a router, in which a pair of high-speed ASICs perform a Ring Access Controller (RAC) function to add or remove data packets to or from a pair of counter rotating optical rings. The first RAC ASIC terminates ring traffic on a xe2x80x9cWestxe2x80x9d side of the ring connection, while the second ASIC terminates the traffic on an xe2x80x9cEastxe2x80x9d side.
The invention provides increased payload capacity as compared to standard interconnect protocols such as Peripheral Component Interconnect (PCI) while also providing for automatic resynchronization upon loss of synchronization at the receive end. Furthermore, the invention lends itself to providing a configurable number of serial links and efficient passing of packet oriented data over the multiple links.